Querying the Internet with PIER

The database research community prides itself on scalable technologies. Yet database systems traditionally do not excel on one important scalability dimension: the degree of distribution. This limitation has hampered the impact of database technologies on massively distributed systems like the Internet. In this paper, we present the initial design of PIER, a massively distributed query engine based on overlay networks, which is intended to bring database query processing facilities to new, widely distributed environments. We motivate the need for massively distributed queries, and argue for a relaxation of certain traditional database research goals in the pursuit of scalability and widespread adoption. We present simulation results showing PIER gracefully running relational queries across thousands of machines, and show results from the same software base in actual deployment on a large experimental cluster. Comments Permission to copy without fee all or part of this material is granted provided that the copies are not made or distributed for direct commercial advantage, the VLDB copyright notice and the title of the publication and its date appear, and notice is given that copying is by permission of the Very Large Data Base Endowment. To copy otherwise, or to republish, requires a fee and/or special permission from the Endowment. Published in Proceedings of the International Conference on Very Large Data Bases, 29th Edition, September 2003, pages 321-332. NOTE: At the time of publication, author Boon Thau Loo was affiliated with the University of California at Berkeley. Currently (April 2007), he is a faculty member in the Department of Computer and Information Science at the University of Pennsylvania. Author(s) Ryan Huebsch, Joseph M. Hellerstein, Nick Lanham, Boon Thau Loo, Scott Shenker, and Ion Stoica This conference paper is available at ScholarlyCommons: http://repository.upenn.edu/cis_papers/326 Querying the Internet with PIER Ryan Huebsch Joseph M. Hellerstein Nick Lanham Boon Thau Loo Scott Shenker Ion Stoica EECS Computer Science Division, UC Berkeley huebsch@,jmh@,nickl@db.,boonloo@,istoica@ cs.berkeley.edu, International Computer Science Institute [email protected]